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ABSTRACT 

In  field  tests  to  compare  the  observability  of  combat  vehicles,  the  test 
designer  must  select  the  optimum  number  of  observation  opportunities  in  order 
to  balance  collecting  enough  data  to  draw  valid  conclusions  against  the  high 
cost  of  supporting  vehicles  and  personnel  at  a  test  site.  The  test  designer, 
however,  generally  lacks  key  parameters  for  the  efficient  design  of  the  test. 

Namely,  the  designer  lacks  the  detection  probabilities  of  the  vehicles  at  each 
range.  The  standard  deviation  of  the  difference  in  detection  probability 
depends  upon  the  detection  probability  itself.  Therefore,  the  test  designer  must 
select  the  number  of  observations  for  each  range  based  upon  the  conservative 
assumption  that  the  probabilities  are  near  50%,  the  probability  for  the 
maximum  standard  deviation.  In  this  paper,  an  iterative  technique  of  test 
design  is  explored  in  order  to  improve  the  efficiency  of  observability  tests. 

THE  EXPERIMENTAL  SITUATION 

Figure  1  illustrates  a  typical  test  setup  for  a  test  of  observability.  Observers  are  stationed 
at  a  fixed  site  and  attempt  to  detect  a  vehicle  in  their  field  of  view.  For  each  observation 
opportunity,  the  test  personnel  record  the  number  of  detections.  In  analyzing  the  data,  the 
analyst  groups  the  observations  into  range  bins  and  compares  the  proportion  of  detections  for 
each  test  vehicle. 

For  such  a  field  test,  the  test  designer  must  select  the  optimum  number  of  observation 
opportunities  at  each  range.  The  designer  must  balance  collecting  enough  data  to  draw  valid 
conclusions  against  the  high  cost  of  supporting  vehicles  and  personnel  at  a  test  site. 

DESIGNING  THE  EXPERIMENT 

The  test  designer  selects  the  number  of  observations  in  order  to  achieve  two  objectives. 
First,  the  number  of  observations  must  be  large  enough  so  that  the  probabilities  of  errors  are 
below  the  desired  level.  And,  second,  the  number  of  observations  must  be  as  small  as  possible  to 
minimize  testing  time  and  cost. 

Table  I  illustrates  the  types  of  errors  that  can  occur  in  drawing  conclusions  fi’om  an 
observation  test.  The  goal  of  an  observation  test  is  to  judge  whether  the  observed  difference  in 
detection  probability,  Pd ,  is  imusual  enough  to  reject  the  null  hypothesis  that  Vehicle  A  and 
Vehicle  B  have  the  same  Pd .  If  the  analyst  concludes  that  the  null  hypothesis  cannot  be  rejected, 
then  either  the  analyst  has  made  a  correct  decision  or  a  Type  II  error.  On  the  other  hand,  a 
decision  to  reject  the  null  hypothesis  will  be  either  correct  or  a  Type  I  error.  In  terms  of 
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countermeasure  efifectiveness,  a  Type  I  error  is  an  erroneous  conclusion  than  an  ineffective 
countermeasure  is  effective,  while  a  Type  n  error  miss  judges  an  effective  countermeasure. 

In  a  previous  paper  [1],  I  discussed  the  number  of  observation  opportunities  required  to 
control  the  probability  of  these  errors.  For  the  case  of  an  average  Pd  of  0.5,  Figure  2  plots  the 
number  of  observations  required  to  maintain  the  probability  of  Type  I  and  Type  11  errors  at  less 
than  5%.  For  example,  85  observations  of  each  vehicle  are  adequate  for  an  observed  0.15 
difference  to  be  significant  with  less  than  5%  chance  of  a  Type  I  error.  On  the  other  hand,  for 
less  that  5%  chance  of  committing  a  Type  II  error  when  the  underlying  probabilities  differ  by 
0.15,  the  requirement  is  316  observation  opportunities  per  vehicle. 

AN  ITERATIVE  TECHNIQUE  FOR  TEST  DESIGN 

An  iterative  technique  can  reduce  the  number  of  observation  opportunities  required  for  a 
test.  Observation  data  is  described  by  the  binary  distribution.  But  the  binary  distribution  has  the 
property  that  the  standard  deviation  depends  upon  the  probability.  If  p  is  the  probability  and  N  is 
the  number  of  trials,  then  the  standard  deviation  of  the  number  of  successes,  o,  is  given  by 


<r  =  -yjNp{\-  p)  (1) 

And  the  standard  deviation  of  the  proportion  of  successes  is  given  by 

a  _  Ip(1-p)  .y. 

V  V  V 

Figure  3  plots  Equation  2  for  an  N  of  100, 200  and  300.  Figure  4  plots  the  same  equation 
normalized  to  its  maximum  at  p  =  0.5. 

Figure  5  further  shows  how  the  number  of  trials  change  with  probability  if  the  standard 
deviation  is  held  fixed.  For  example,  in  comparison  to  a  probability  of  0.5,  only  60%  as  many 
trials  are  need  at  a  probability  of  0. 1 . 

To  use  this  property  of  the  binary  distribution,  the  test  designer  modifies  the  test  matrix 
as  the  test  progresses.  Early  test  results  yield  estimates  of  the  probability  of  detection  for  each 
vehicle  at  each  range.  Based  on  these  estimates,  the  test  designer  reallocates  the  number  of 
observations  among  the  ranges,  improving  the  efficiency  of  the  test  design.  Figure  5  illustrates  a 
sample  of  the  improveuKnt  in  efficiency  that  this  technique  can  achieve.  Initially  the  test 
designer  would  have  selected  the  number  of  observations  at  each  test  range  by  assuming  the 
worst-case  value  of  Pd=  0.5.  But  as  the  test  progresses,  the  designer  would  use  estimates  of  Pd 
from  the  early  measures  to  redesign  the  test.  For  example,  at  range  20,  the  designer  would  reduce 
the  number  of  observations  from  3 14  to  25 1.  Overall  in  this  sample  test,  the  test  designer  would 
reduce  the  number  of  observations  from  1570  to  1 193,  a  reduction  of  24%. 


CONCLUSION 


In  this  paper,  I  have  presented  an  iterative  technique  of  using  early  test  results  in 
observation  test  to  improve  the  overall  efficiency  of  the  test  design. 
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Possible  Outcomes  of  Hypothesis  Testing 


ts  Vehicle  A  Less  Detectable 
Than  VeMcie  B? 


Decision: 

No 

Yes 

Accept  Null  Hypothesis 

C  orrect 

T ype  II  Error 

Accept  Alternative  Hypothesis 

Type  1  Error 

C  orrect 

Table  1.  Definitions  of  Type  I  and  Type  11  errors. 


Test  with  Fixed  Observers 


Fixed  Observers 


Figure  1.  Experimental  setup  for  an  observer  test. 


Number  of  Opportunities 
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Figure  2.  Number  of  observation  opportunities  required  to  meet  test  criteria. 


Initial  and  Optimum  Number  of  Observations  per  Vehicle 
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Figure  5.  An  example  of  iterative  design  of  an  observer  test. 
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standard  Deviation  vs.  Probability 


Standard 


Figure  3.  Standard  deviation  of  a  binary  distribution  for  100,  200  and  300  trials. 


Number  of  Observation  for  Constant  Standard  Deviation 

N(%ofNfora.S) 


Figure  4.  Standard  deviation  of  a  binary  distribution  normalize  to  probability  of  0.5. 
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